Oracle Inequalities and Adaptive Rates

We have previously seen how sieve estimators give rise to rates of convergence to the Bayes risk by performing empirical risk minimization over Hk(n), where (Hk)k ≥ 1 is an increasing sequence of sets of classifiers, and k(n) → ∞. However, the rate of convergence depends on k(n). Usually this rate is chosen to minimize the worst-case rate over all distributions of interest. However, it would be nice if we could automatically get a faster rate of convergence when the distribution is more favorable. Since we don’t know whether our distribution is worst-case or not a priori, we don’t know how to choose k(n), and adaptive rates of convergence are not possible with sieve estimators. Adaptive rates are possible, however, using another learning strategy called penalized empirical risk minimization. With this approach we can prove a so-called “oracle inequality” that expresses the ability of penalized ERM to automatically select a classifier of the appropriate complexity so as to achieve improved rates of convergence, even when the best model class Hk depends on some unknown property of the distribution. In these notes we will consider oracle inequalities in the context of dyadic decision trees and of general VC classes. In the latter case, penalized empirical risk minimization is known as structural risk minimization.